REMARKS 



Claims 1-3, 5-6, 9, 11-12, 14, 16, 26-37 and 39-42 were presented for examination 
and were rejected. Applicants thank the Examiner for examination of the claims pending in 
this application and addresses the Examiner's comments below. 

Reconsideration of the application in view of the following remarks is respectfully 
requested. 

In an office action mailed May 18, 2007, the Examiner rejected the claims 1-3, 5-6, 9, 
11-12, 14, 16, 26-37, 39-42 under 35 U.S.C. § 103(a) as being unpatentable over Rubin et al. 
("Rubin," US Patent Publication No. 2002/0099552) in view of Heck et al. ("Heck," "A 
Survey of Web Annotation Systems"), Mitchell et al. ("Mitchell", US Patent No. 5,857,099) 
and Balabanovic ("Balabanovic," US Patent No. 6,624,826). 

Applicants respectfully submit that use of the Balabanovic reference is improper as a 
basis of a section 103 rejection. The Balabanovic reference is a reference commonly owned 
by the assignee of the present application at the time the claimed invention was made. The 
Balabanovic reference at most quahfies as prior art under 102(e) since it was published after 
the present application was filed, but has a filing date before the filing date of the present 
application. The Balabanovic references is NOT prior art under 1 02 (a), (b), (c) or (d). Thus, 
if the Balabanovic reference is not considered prior art for section 103, the Examiner has not 
established a prima facie case of obviousness, and the claimed invention is patentably distinct 
over the art of record. Thus, Applicants request allowance of the present application. 
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In view of 35 USC § 103(c), Applicants respectfiilly submit that the Balabanovic 
reference cannot be used to preclude patentability of the claimed invention because per that 
section: 

(1) Subject matter developed by another person, which qualifies as 
prior art only under one or more of subsections (e), (f), and (g) of 

section 102 of this title, shall not preclude patentability under this 
section where the subject matter and the claimed invention were, at the 
time the claimed invention was made, owned bv the same person or 
subject to an obligation of assignment to the same person . (Emphasis 
added) 

Here, the application for the claimed invention was filed on January 9, 2002, before 
the issue date of Balabanovic of September 23, 2003. Therefore, Balabanovic at best is 
citeable art under 35 USC § 102(e). The Balabanovic reference was assigned to Ricoh Co., 
Ltd, Tokyo, Japan in an assignment recorded December 13, 1999 and recorded with the U.S. 
Patent Office at Reel 010456, Frame 0160. A copy of the assignment history for the 
Balabanovic reference is attached as Exhibit A for Examiner's reference. The present 
application with the claimed invention was assigned to the same Ricoh Co., Ltd, Tokyo, 
Japan in an assignment recorded on March 19, 2002 at Reel 012722, Frame 0848. A copy of 
this assignment is attached as Exhibit B for Examiner's reference. The title to both the 
Balabanovic reference and the present application currently remain assigned to Ricoh Co., 
Ltd, Tokyo, Japan. 

Thus, the subject matter of the Balabanovic reference and the present application with 
the claimed invention were, at the time the present application with the claimed invention 
was made, owned by the same person or subject to an obligation of assignment to the same 
person. Under 35 USC § 103(c) the subject matter of Balabanovic cannot preclude 
patentability of the claimed invention. Therefore, it is respectfully submitted that the basis 
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for the rejection to claims 1-3, 5-6, 9, 1 1-12, 14, 16, 26-37, 39-42 is improper and should be 
withdrawn. 

Additionally, the claimed invention provides a number of advantages which are not 
disclosed or suggested by Rubin, Heck and Mitchell, either alone or in combination. 

Claim 1 is representative of the independent claims and is directed to the direct multi- 
modal annotation of objects. This is reflected in the language "direct annotation creation 
module." The process of labeling an annotation is one of the key differences between the 
claimed invention and prior art. This is set forth in the claim language "a plurality of 
different visual notations each comprising a text or a graphic image." In particular, the 
claimed invention compares input with stored exemplars, labels the location of input with 
stored exemplars and then labels/links the input with the corresponding exemplar label. This 
is set forth in the claim language "the audio vocabulary comparison module receiving audio 
input and finding a corresponding one of the plurality of different visual notations;" 
"automatically creating an annotation object, independent from the image, that associates the 
input audio signal, the location and one of the plurality of different visual notations," 
respectively. Furthermore, the dependent claims provide for automatic creation of a new 
exemplar and associated label if there is no sufficiently good match for the input. 

Hence, the claimed invention compares the input to a stored vocabulary with the 
audio vocabulary comparison module. This is not full-on speech recognition, but comparison 
between an input and stored exemplars. Since there are likely to be only a limited number of 
labels that are reused, this is much faster and accurate than traditional speech recognition. 
Furthermore, even if Rubin, Heck and Mitchell were combined, they would not yield the 
claimed invention where there is a relationship between the label and the image/position and 
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the audio. There is no teaching or suggestion of such a three way relationship in Rubin, 
Heck or Mitchell, either alone or in combination. 

Second, the claimed invention provides distinct labels or visual notations each 
comprising a text or a graphic image. Assuming a matching exemplar is foimd, the claimed 
invention labels the input with the label associated with that exemplar. For example, 
particular labels may be text or phases describing a portion of the image such as "Uncle 
Fred" for one location and "Auntie June" for another location. This differs dramatically from 
the cited references where the label is simply the same generic audio icon for all annotations, 
or simply differences in color. 

Third, the claimed invention simplifies creation of new labels by using a limited 
vocabulary. If a good match for an annotation is not found, the user is given the option to 
either create a new exemplar (and label) or select an existing exemplar. In the second case, 
the audio input is used as an additional training point for that exemplar (meaning the 
accuracy in recognizing the input will increase for the next time that similar audio input is 
encountered). 

Independent claim 9 includes limitations similar to claim 1 . Thus, all arguments 
advanced above with respect to claim 1 also apply to claim 9. Hence, claim 9 is patentably 
distinguishable over the cited references, both alone and in combination. 

As claims 2, 3, 5 and 6 depend from claim 1, all arguments advanced above with 

respect to claim 1 are hereby incorporated so as to apply to claims 2, 3, 5 and 6. Hence, 
claims 2, 3, 5 and 6 are patentably distinguishable over the cited references, both alone and in 
combination. 
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Claim 26 is an independent method claim that includes limitations similar to those 
recited above for claim 1 . Thus, all arguments advanced above with respect to claim 1 also 
apply to claim 26. Hence, claim 26 is patentably distinguishable over the cited references, 
both alone and in combination. 

As claims 11, 12, 14, 16 and 27-34 depend from claim 26, all arguments advanced 
above with respect to claim 1 are hereby incorporated so as to apply to claims, 11, 12, 14, 16 
and 27-34. Hence, claims 11, 12, 14, 16 and 27-34 are patentably distinguishable over the 
cited references, both alone and in combination. 

Claims 35-37 and 39-42 were previously presented. Applicants submit that 
combination of Rubin, Heck and Mitchell fails to disclose the recited steps of "determining 
whether the annotation includes text," "retrieving a text annotation for the selected visual 
notation," or "displaying the retrieved text with the image." Thus, this combination does not 
distinguish between different annotations with different text and audio associated with each 
annotation. Since, at best, only one annotation or annotations of different colors, neither of 
which can have both audio and/or text associated therewith, are disclosed by the 
combination. Applicants submit that these annotation display methods arc unique to the 
annotation created and provided by the claimed invention and are not taught or suggested by 
the references, both alone and in combination. 

CONCLUSION 

In sum, Applicants respectfully submit that claims 1-3, 5, 6, 9, 11, 12, 14, 16, 26-37, 
and 39-42, as presented herein, are patentably distinguishable over all of the art of record. 
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Therefore, Applicants request reconsideration of the basis for the rejections to these claims 



and request allowance of the claims. 



In addition, Applicants respectfully invite Examiner to contact Applicants' 



representative at the number provided below if Examiner believes it will help expedite 



furtherance of this application. 



Respectfully Submitted, 
GREGORY J. WOLFF, et al. 



Greg T. Sueoka, Reg. No.: 33,800 
Fenwick & West LLP 
Silicon Valley Center 
801 California Street 
Mountain View, CA 94041 
Tel.: (650)335-7194 
Fax: (650)938-5200 



Date: 
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